build: Add spark-4.1 profile and shims by manuzhang · Pull Request #2829 · apache/datafusion-comet

manuzhang · 2025-11-27T08:16:01Z

Which issue does this PR close?

First step of #2792.

Rationale for this change

What changes are included in this PR?

Add spark-4.1 profile with minor shim version spark-4.1
Move src/main/spark-4.0 to src/main/spark-4.x for common shim classes in spark-4.0 and spark-4.1.
Add CometSumShim and ShimSQLConf for spark-4.0 and spark-4.1 specific shims respectively
Add MapStatusBuilder.scala to access org.apache.spark.scheduler.MapStatus in java. MapStatus has added a constructor argument in Spark 4.1, and only kept compatibility for scala codes.

Summary of Changes in 4.1.1.diff (generated by Copilot)

This patch updates the Comet project to support Spark 4.1. The changes largely involve build configuration updates, integration into Spark's session initialization, and extensive test exclusions for features not yet supported or behaving differently in Comet.

Build Configuration

pom.xml:
- Updated spark.version.short to 4.1.
- Added dependency for comet-spark-spark4.1.
sql/core/pom.xml:
- Added comet-spark-spark4.1 dependency to the SQL core module.

Core Spark Integration

SparkSession.scala:
- Added isCometEnabled check (defaults to true if ENABLE_COMET env var is not set or true).
- Added loadCometExtension to automatically inject org.apache.comet.CometSparkSessionExtensions when enabling Comet.
SparkPlanInfo.scala:
- Added support for CometScanExec to properly extract metadata for Spark UI/history.

Test Infrastructure

IgnoreComet.scala:
- Introduced new Scalatest tags: IgnoreComet, IgnoreCometNativeIcebergCompat, IgnoreCometNativeDataFusion, IgnoreCometNativeScan.
- Added IgnoreCometSuite trait to disable entire test suites when Comet is enabled.

Test Exclusions

A large number of existing Spark SQL tests have been modified to skip when Comet is enabled. Common reasons cited in the diffs include:

Unsupported Features:
- SubqueryBroadcastExec ([INTERNAL_ERROR] Custom columnar rules cannot transform shuffle node to something else #1737, Support SubqueryBroadcastExec in Comet #242)
- Spill support in SortMergeJoin
- Native CometSort peak execution memory updates
- Datetime rebase mode
- Parquet column index
- RLE encoding, DELTA encoding
- Variant types (issues/2209)
Behavioral Differences:
- Explain output differences
- Shuffle partition size and metrics changes
- Exception handling (e.g. Cast context)
Pending Fixes:
- References to various GitHub issues (e.g., Investigate failing tests with Spark 4.0.0 #1948, Fix listagg-collation.sql test in Spark 4.0.0 #1947, [Spark 4.0.0] Cast in ANSI mode uses wrong exception class #2218, Fix Spark 4.0 sql tests #551).

Modified Test Files

The following logical areas of tests had significant exclusions:

Adaptive Query Execution (AQE)
Join Suites (DataFrameJoin, JoinHint, JoinSuite)
Parquet Data Source (Filter, Encoding, Query, Schema)
SQL Query execution and metrics
Collation support

How are these changes tested?

Added UTs.

codecov-commenter · 2025-11-27T08:39:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.02%. Comparing base (f09f8af) to head (26a609e).
⚠️ Report is 918 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2829      +/-   ##
============================================
+ Coverage     56.12%   60.02%   +3.90%     
- Complexity      976     1469     +493     
============================================
  Files           119      176      +57     
  Lines         11743    16174    +4431     
  Branches       2251     2682     +431     
============================================
+ Hits           6591     9709    +3118     
- Misses         4012     5113    +1101     
- Partials       1140     1352     +212

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

manuzhang · 2025-12-23T00:22:33Z

@andygrove @coderfender Please help review. Do we need to pass all tests now?

andygrove · 2026-01-05T21:02:58Z

Test failure:

CometExec3_4PlusSuite:
- subquery limit: limit with offset should return correct results (565 milliseconds)
- offset (602 milliseconds)
- test BloomFilterMightContain can take a constant value input (231 milliseconds)
- test NULL inputs for BloomFilterMightContain (251 milliseconds)
- test BloomFilterMightContain from random input *** FAILED *** (511 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 41.0 failed 1 times, most recent failure: Lost task 1.0 in stage 41.0 (TID 102) (localhost executor driver): org.apache.comet.CometNativeException: assertion `left == right` failed: Unsupported BloomFilter version: 2, expecting version: 1
  left: 2
 right: 1
        at comet::errors::init::{{closure}}(__internal__:0)
        at std::panicking::panic_with_hook(__internal__:0)
        at std::panicking::panic_handler::{{closure}}(__internal__:0)
        at std::sys::backtrace::__rust_end_short_backtrace(__internal__:0)
        at __rustc::rust_begin_unwind(__internal__:0)
        at core::panicking::panic_fmt(__internal__:0)
        at core::panicking::assert_failed_inner(__internal__:0)
        at core::panicking::assert_failed(__internal__:0)
        at <datafusion_comet_spark_expr::bloom_filter::spark_bloom_filter::SparkBloomFilter as core::convert::From<&[u8]>>::from(__internal__:0)

andygrove · 2026-01-05T21:03:43Z

@andygrove @coderfender Please help review. Do we need to pass all tests now?

Yes, we either need tests to pass, or we can potentially disable some specific tests by updating the diff file, and file issues to resolve those failures.

manuzhang · 2026-01-06T02:05:48Z

@andygrove Sure, since Spark 4.1.1 is to be released soon, I will check again after upgrading to 4.1.1

andygrove · 2026-01-30T16:11:25Z

Thanks for continuing to work on this @manuzhang!

We found some issues with native columnar to row, so we had to disable it in #3348, and this affects the golden files, so you may need to regenerate them after this PR gets merged.

manuzhang · 2026-02-02T03:32:45Z

Thanks @andygrove, I'm trying to catch up recent changes on main branch including sql file tests.

Co-authored-by: Andy Grove <agrove@apache.org>

Signed-off-by: manuzhang <owenzhang1990@gmail.com>

manuzhang marked this pull request as draft November 27, 2025 08:28

manuzhang force-pushed the spark-4.1 branch 4 times, most recently from 31cf1ca to 88f8b68 Compare November 28, 2025 04:33

manuzhang force-pushed the spark-4.1 branch from 88f8b68 to 89577ad Compare December 22, 2025 02:40

manuzhang marked this pull request as ready for review December 22, 2025 02:43

manuzhang force-pushed the spark-4.1 branch 4 times, most recently from 6d6dfb4 to 2956aac Compare December 22, 2025 10:50

manuzhang mentioned this pull request Dec 23, 2025

feat: upgrade Spark 4.x support from 4.0.1 to 4.1.0 [WIP] #2961

Draft

manuzhang force-pushed the spark-4.1 branch 3 times, most recently from 4207552 to e65600c Compare December 25, 2025 09:42

manuzhang force-pushed the spark-4.1 branch 6 times, most recently from 524d90a to d3eed18 Compare January 14, 2026 16:15

manuzhang force-pushed the spark-4.1 branch 4 times, most recently from a2e6949 to 1fa26b6 Compare January 29, 2026 10:10

manuzhang force-pushed the spark-4.1 branch 7 times, most recently from be131a2 to b8fa052 Compare January 30, 2026 07:40

manuzhang force-pushed the spark-4.1 branch 4 times, most recently from 2cfe6cc to 08fa357 Compare February 2, 2026 02:18

manuzhang force-pushed the spark-4.1 branch 2 times, most recently from 44c5a19 to 90a9b83 Compare February 3, 2026 04:46

manuzhang closed this Feb 3, 2026

manuzhang reopened this Feb 3, 2026

manuzhang and others added 7 commits February 3, 2026 14:46

build: Add spark-4.1 profile and shims

b937f3a

Co-authored-by: Andy Grove <agrove@apache.org>

update for bloom filter v2

2b4f86e

Skip tests for Spark 4.1

09cf716

Fix PlanStabilitySuite

6dbdc7e

Skip more tests

2fc481d

update sql golden files and diff

fb54252

remove some maven caches

c7a6c12

manuzhang force-pushed the spark-4.1 branch 3 times, most recently from 54577cf to 26a609e Compare February 3, 2026 14:42

manuzhang mentioned this pull request Feb 4, 2026

Investigate failing tests with Spark 4.1.1 #3384

Open

Update diff and Skip CometSqlFileTestSuite

780975b

Signed-off-by: manuzhang <owenzhang1990@gmail.com>

manuzhang force-pushed the spark-4.1 branch from 26a609e to 780975b Compare February 4, 2026 04:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Add spark-4.1 profile and shims#2829

build: Add spark-4.1 profile and shims#2829
manuzhang wants to merge 8 commits intoapache:mainfrom
manuzhang:spark-4.1

manuzhang commented Nov 27, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Nov 27, 2025 •

edited

Loading

Uh oh!

manuzhang commented Dec 23, 2025

Uh oh!

andygrove commented Jan 5, 2026

Uh oh!

andygrove commented Jan 5, 2026

Uh oh!

manuzhang commented Jan 6, 2026

Uh oh!

andygrove commented Jan 30, 2026

Uh oh!

manuzhang commented Feb 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

manuzhang commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Summary of Changes in 4.1.1.diff (generated by Copilot)

Build Configuration

Core Spark Integration

Test Infrastructure

Test Exclusions

Modified Test Files

How are these changes tested?

Uh oh!

codecov-commenter commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

manuzhang commented Dec 23, 2025

Uh oh!

andygrove commented Jan 5, 2026

Uh oh!

andygrove commented Jan 5, 2026

Uh oh!

manuzhang commented Jan 6, 2026

Uh oh!

andygrove commented Jan 30, 2026

Uh oh!

manuzhang commented Feb 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

manuzhang commented Nov 27, 2025 •

edited

Loading

codecov-commenter commented Nov 27, 2025 •

edited

Loading